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MIIC Project 


A ROSES ACCESS 2013 project (2 years) to 
deploy web services at NOAA NCDC and NASA 
ASDC to improve access to select data 
products in support of intercalibration and 
intercomparison studies 
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Ml 1C Objectives 


Access matched observations (time, view angles) from multiple 
satellite instruments within large datasets distributed across 
multi-agency data centers 

Filter large data products within server-side functions at 
remote centers prior to network transmission 

Support GSICS, Cal-Val and CLARREO teams 

Demonstrate benefit of OPeNDAP server-side functions 

Demonstrate successful deployment of a NOAA and NASA 
collaborative distributed data system 

Demonstrate feasibility of accessing remote hyperspectral 
OSSE datasets for comparison with observations 



Future MIIC Opportunities 


Deploy MIIC-OPeNDAP module at GSFC LAADS, access LI and L2 MODIS data 
and reprocessed VIIRS SDR and EDR (Atmospheres) (£. Masuoka) 

Deploy MIIC-OPeNDAP module at GSFC DAAC, access LI and L2 AIRS data 
and reprocessed CRIS data (S. Kempler) 

Deploy MIIC OPeNDAP server(s) at EUMETSAT, access Metop-A,B (P. Miu, T. 
Hewison) 

Provide MIIC link on the Earth Data Web portal ( http://earthdata.nasa.gov ) 

Serve as basis for Reference Intercalibration architecture for CLARREO 
Pathfinder (2016 -2021) 


Ml 1C Infrastructure to Support CLARREO Pathfinder 

Reference Intercalibration 
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Data Products for Initial MIIC Deployment 


Product 

Center 

Type 

Temporal 

Coverage 

Size 

CERES L2 

ASDC 

Footprints 

hour 

50 MB 

CALIPSO LI 

ASDC 

Profile 

orbit 

500 MB 

CALIPSO L2 

ASDC 

Layer, Profile, and VFM 

orbit 

40-108 MB 

CRISSDR 

NCDC 

Spectra 

6.8 min 

257 MB 

ATMS SDR 

NCDC 

Swath scans 

7 min 

2.6 MB 

VIIRS SDR 

NCDC 

Swath scans 

5 min 

350 MB 

VIIRS EDR 

NCDC 

Aerosol, Cloud, Ice, Land, Sea 
Surface,... 14 types 

~ 5.8 min 

1.1-9 MB 

GOES13 

NCDC 

Full disk scans 

15 min. 

2.7 GB 


• Sizes are approximate; temporal and spatial coverage vary with product type 

• Geolocation and science data may be in separate files at NCDC (use OPeNDAP NcML) 

• VIIRS channels stored in separate files 











Ml 1C Use Cases 


Intercalibration (LEO-GEO and LEO-LEO) ✓ 

■ Acquire and analyze matched data using the MIIC event predictor instrument models 

■ Use MIIC OPeNDAP server-side functions to acquire, filter, and grid LI or L2 data 

■ Demonstrated capabilities with SCIAMACHY, MODIS, VIIRS, GOES and CERES 

Intercomparison ( support NPP Validation study) ✓ 

■ Acquire matched NPP and CALIPSO data 

■ Data used for offline validation of CriMSS (T, H 2 0) using RTM (X. Liu) 

■ Acquire CrIS SDR, ATMS SDR, VIIRS SDR, VIIRS EDR, CriMSS EDR, CALIPSO LI 
& L2 data for matched events 

Surface Site Data Mining ✓ 

Acquire data over geographic surface sites (eg., Libya-4, Dome C) 

■ Use MIIC OPeNDAP server-side functions to acquire, filter, and grid LI or L2 data 

■ For CERES and CALIPSO use ASDC PostGIS to speed up predictions 

OSSE Data Access (support OSSE vs. SCIAMACHY analysis) 

Demonstrate efficient access to large hyperspectral multi-year OSSE datasets 
Develop spectral resample and spatial average server-side functions 

■ OSSE vs. SCIAMACHY offline analysis (Y Roberts) 


What can you do with MIIC? 


• Event Prediction ^ 

• Acquisition of matched events ✓ 

• Analysis and visualization of events 



All products staged on ASDC servers, surface site (10 degree box at 75 W), data volumes are approximate 
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NOAA - NASA Collaborative Deployment 



30 Day Cache w/ OPeNDAP access 


S-NPP MIIC 30 Day Cache 


Index of ftp://ftp-npp.class.ngdc.noaa.gov/20150114/ 

££ Up to higher level directory 


Name 

Size Last Modified 

Q ATMS- SDR 

1/14/15 

8:00:00 AM 

Q ATMS-TDR 

1/14/15 

8:15:00 AM 

□ CRIS-SDR 

1/14/15 

10:01:00 AM 

CD OMPS-EDR 

1/14/15 

9:15:00 AM 

fiiOMPS-IP 

1/14/15 

9:45:00 AM 

CD omps-rdr 

1/14/15 

9:46:00 PM 

CD OMPS-SDR 

1/14/15 

9:30:00 AM 

□ VIIRS-EDR 

1/14/15 

1:12:00 PM 

□ VIIRS-IPNG 

1/14/15 

2:00:00 PM 

□ VIIRS-SDR 

1/14/15 

3:49:00 PM 

Bviirsi-edr 

1/14/15 

12:17:00 PM 


Submitted request to NOAA for NPP file types 
ATMS, VI IRS, and CRIS products (19 TB/mo.) 
Current ftp server not setup for OPeNDAP, files 
need to be untarred and uncompressed, 
ftp://ftp-npp.class.ngdc.noaa.gov/ 

NCDC to stand up dedicated 30 day MIIC cache 
with OPeNDAP access (J. Morris , L. Crandall) 


Index of ftp://ftp-npp.class.ngdc.noaa.gov/20150114/VIIRS-SDR/ 

££ Up to higher level directory 


Name 

Size 

Last Modified 

^ VIIRS-Day-Night-Band-SDR 


1/15/15 

3:47:00 PM 

Q VIIRS-Day-Night-Band-SDR-Ellipsoid-Geo 


1/15/15 

3:46:00 PM 

CD VIIRS-Image-Bands-SDR-Ellipsoid-Ceo 


1/15/15 

3:46:00 PM 

CD VIIRS-Image-Bands-SDR-Ellipsoid-Terrain-Corrected-Geo 


1/15/15 

3:50:00 PM 

CD VIIRS- Imagery- Band -01 -SDR 


1/15/15 

3:48:00 PM 

CD VIIRS-lmagery-Band-02-SDR 


1/15/15 

3:48:00 PM 

CD VIIRS -Imagery- Band -03 -SDR 


1/15/15 

3:50:00 PM 

CD VIIRS-lmagery-Band-04-SDR 


1/15/15 

3:46:00 PM 

□ VIIRS-lmagery-Band-05-SDR 


1/15/15 

3:46:00 PM 

CD VIIRS-Moderate-Bands-SDR-Geo 


1/15/15 

3:49:00 PM 

CD VIIRS-Moderate-Bands-SDR-Terrain-Corrected-Geo 


1/15/15 

3:45:00 PM 

CD VIIRS-Moderate-Resolulion-Band-01-SDR 


1/15/15 

3:51:00 PM 

CD VIIRS-Moderate-Resolution-Band-02-SDR 


1/15/15 

3:48:00 PM 

CD VIIR5-Moderate-Resolution-Band-03-SDR 


1/15/15 

3:45:00 PM 

CD VIIRS-Moderate-Resolution-Band-04-SDR 


1/15/15 

3:50:00 PM 

CD VIIRS-Moderate-Resolution-Band-05-SDR 


1/15/15 

3:45:00 PM 

Cl VIIRS-Moderate-Resolulion-Band-06-SDR 


1/15/15 

3:47:00 PM 

CD VIIRS-Moderate-Resolution-Band-07-SDR 


1/15/15 

3:49:00 PM 

Cj VIIRS-Moderate-Resolution-Band-08-SDR 


1/15/15 

3:44:00 PM 

CD VIIRS-Moderate-Resolution-Band-09-SDR 


1/15/15 

3:51:00 PM 

CD VIIRS-Moderate-Resolution-Band-10-SDR 


1/15/15 

3:45:00 PM 

CD VIIRS-Moderate-Resolution-Band-11-SDR 


1/15/15 

3:44:00 PM 

CD VIIRS-Moderate-Resolution-Band- 12-SDR 


1/15/15 

3:48:00 PM 

CD VIIRS-Moderate-Resolution-Band- 13-SDR 


1/15/15 

3:49:00 PM 

CD VIIRS-Moderate-Resolution-Band-14-SDR 


1/15/15 

3:51:00 PM 

CD VIIRS-Moderate-Resolution-Band -15 -SDR 


1/15/15 

3:49:00 PM 

^ VIIRS-Moderate-Resolution-Band-16-SDR 


1/15/15 

3:47:00 PM 
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MIIC Web Page Demo 


Intercalibration Plan 


Event Prediction Options Data Collection Options 


Plan Name 
Owner 

Target Data Collection 
Reference Data Collection 
Begin date 
End date 


CERES_Aqua-NPP_VIIRS-dsrun 

(jon 

npp_viirs_sdr_subsample 

ceres_ssf_l2_aqua 

[01/01/2014 

[02/01/2014 


Save Plan 


Home 


View Loo s 


Intercalibration Plan 

Intercalibration Plan Event Prediction Options 


I 




No Data Collection 


2D Histogram Averages 


Full-resolution N-Tuples 


Target Variables BrightnessTemperature_l4_Avg, Latitude_Sub, Longitude Sub 


Reference Variables T OA_and_Surface_Fluxes_CERES_SW_TOA_flux upwards, 

TOA_and_Surface_Fluxes_CERES_net_SW_surface_flux Model_A, 

Time_and_Position_Colatitude_of_CERES_FOV_at_surface, 

Time_and_Position_Longitude_of_CERES_FOV_at_surface 


► Advanced Options 


Save Plan 

_r 


Home 


View Loo s 
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Intercalibration Plan 



View Lo os 


▼ Target variable dimensions ▼ 

▼ Reference variable dimensions ▼ 


User creates 
ICPIans to predict, 
acquire, and 
analyze data 




MIIJ^ Multi Instrument Inter-Calibration 


Welcome! jon Login Logout 


Plan: CERES_Aqua-NPP_VIIRS-dsrun 
Current State: ACQUIRING_DATA 


Start Execution 


Show 

10 i entries 






Search: 



Begin 

- End 

0 Lat South 

0 Lat North 

0 Lon West 

0 Lon East 

0 State 

0 

© 

Fri, 24 Jan 2014 13:59:35 GMT 

Fri, 24 Jan 2014 14:28:38 GMT 

-89.00324197697755 

-2269774608270527 

-179.9761271374854 

179.96991327080818 

DATA_ACQUIRED 


© 

Mon, 27 Jan 2014 03:12:34 GMT 

Mon, 27 Jan 2014 03:44:32 GMT 

-40.79488007908672 

71.49217490581624 

93.67320698173019 

176.74090509262373 

DATA_ACQUIRED 



Mon, 27 Jan 2014 04:35:32 GMT 

Mon, 27 Jan 2014 05:24:59 GMT 

-88.99962790759989 

69.95656783254812 

0.37544132975204914 

-0.05534390474684869 

DATA_ACQUIRED 


® 

Mon, 27 Jan 2014 06:16:47 GMT 

Mon, 27 Jan 2014 06:27:00 GMT 

-88.9996738904554 

-60.816009775757145 

0.07700780640099117 

-0.6418072707076021 

DATA_ACQUIRED 


® 

Wed, 29 Jan 2014 19:1321 GMT 

Wed, 29 Jan 2014 20:01:43 GMT 

-88.99989074878894 

72.18100616579439 

-179.34322853682022 

179.8608894261324 

DATA_ACQUIRED 


® 

Wed, 29 Jan 2014 20:52:28 GMT 

Wed, 29 Jan 2014 21:41:41 GMT 

-88.99990601738796 

68.98370623285292 

-179.65961033892637 

179.66108203012388 

DATA_ACQUIRED 


Showing 21 to 26 of 26 entries 
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Event Prediction results depicted on 
Google Maps 

Need to develop analysis and 
visualization capabilities for single 
events and collections of events 



TLEs Available for MIIC Event Prediction 

(red implemented) 


Can run predictions for 
any spacecraft with TLEs 
from Space-Track.org 

Current plans to access 
only instrument data 
hosted at ASDC or NCDC 

User can run predictions 
w/o accessing data 

Extend list of satellites in 
current deployment? 

Run 1C and access only 
data from 1 satellite 

Extended CLARREO 
algorithm (C. Roithmayr) 


http://www-angler.larc.nasa.gov/cgi-bin/site/showdoc? 

27386 

"EnviSAT" 

"2013-06-20 08:05:02" 

mnemonic=TI_E&c=satellite-list 

27421 

"SPOT-5" 

"2013-06-20 08:05:02" 




27424 

"Aqua" 

"2013-06-20 08:05:02" 

CAT# 

NAME 

TLE_LAST_UPDATED 

27453 

"NOAA-17" 

"2013-06-20 08:05:02" 

13367 

"Landsat 4" 

"2013-06-20 08:05:02" 

27509 

"Meteosat-8 (MSG-l)" "2013-06-20 08:05:02' 

14780 

"Landsat 5" 

"2013-06-20 08:05:02" 

27525 

"Kalpana-1" 

"2013-06-20 08:05:02" 

15354 

"ERBS" 

"2013-06-20 08:05:02" 

27642 

"ICESat" 

"2013-06-20 08:05:02" 

15427 

"NOAA-09" 

"2013-06-20 08:05:02" 

28054 

"DMSP 5D-3 F16 (USA 172) 

16969 

"NOAA-IO" 

"2013-06-20 08:05:02" 

28376 

"Aura" 

"2013-06-20 08:05:02" 

19531 

"NOAA-11" 

"2013-06-20 08:05:02" 

28451 

"FengYun 2C" 

"2013-06-20 08:05:02" 

21140 

"Meteosat-5" 

"2013-06-20 08:05:02" 

28622 

"MTSAT-1R" 

"2013-06-20 08:05:02" 

21263 

,, NOAA-12" 

"2013-06-20 08:05:02" 

28622 

"MTSAT-1R/Aqua" "2013-06-20 08:05:02" 

21574 

"ERS-1" 

"2013-06-20 08:05:02" 

28654 

"NOAA-18" 

"2013-06-20 08:05:02" 

23051 

"GOES-8" 

"2013-06-20 08:05:02" 

28912 

"Meteosat-9 (MSG-2)" "2013-06-20 08:05:02' 

23455 

"NOAA-14" 

"2013-06-20 08:05:02" 

28912 

" Meteosat-9/Terra (MSG-2)" "2013-06-20 08:05:02' 

23522 

"GMS-5" 

"2013-06-20 08:05:02" 

28912 

"Meteosat-9/Aqua (MSG-2)" "2013-06-20 08:05:02' 

23560 

"ERS-2" 

"2013-06-20 08:05:02" 

28937 

"MTSAT-2" 

"2013-06-20 08:05:02" 

23581 

"GOES-9" 

"2013-06-20 08:05:02" 

29107 

"CloudSAT" 

"2013-06-20 08:05:02" 

23940 

"TOMS-EP" 

"2013-06-20 08:05:02" 

29108 

"Calipso" 

"2013-06-20 08:05:02" 

24786 

"GOES-10" 

"2013-06-20 08:05:02" 

29155 

"GOES-13" 

"2013-06-20 08:05:02" 

24883 

"OrbView-2" 

"2013-06-20 08:05:02" 

29499 

"MetOp-A" 

"2013-06-20 08:05:02" 

24932 

"Meteosat-7" 

"2013-06-20 08:05:02" 

29522 

"DMSP5D-3 F17 

(USA 173)" "2011-11-15 08:12:53' 

25063 

"TRMM/VIRS" 

"2013-06-20 08:05:02" 

29640 

"FengYun 2D" 

"2013-06-20 08:05:02" 

25260 

"SPOT-4" 

"2013-06-20 08:05:02" 

32382 

"RadarSat-2" 

"2013-06-20 08:05:02" 

25338 

"NOAA-15" 

"2013-06-20 08:05:02" 

33463 

"FY2E" 

"2012-07-12 15:57:39" 

25682 

"Landsat 7" 

"2013-06-20 08:05:02" 

33492 

"GOSAT" 

"2013-06-20 08:05:02" 

25789 

"QuikSCAT" 

"2013-06-20 08:05:02" 

33591 

"NOAA-19" 

"2013-06-20 08:05:02" 

25991 

"DMSP 5D-3 F15 (USA 147) 

35951 

"DMSP5D-3 F18 

(USA 210) 

25994 

"Terra" 

"2013-06-20 08:05:02" 

36411 

"GOES-15" 

"2013-06-20 08:05:02" 

26352 

"GOES-11" 

"2013-06-20 08:05:02" 

37838 

"Megha-Tropiques""2013-06-20 08:05:02" 

26536 

"NOAA-16" 

"2013-06-20 08:05:02" 

37849 

"Suomi National Polar Orbiting Partnership" 

26871 

"GOES-12" 

"2013-06-20 08:05:02" 

38552 

"2013-06-20 08:05:02" 

"Meteosat-10 (MSG-3)" "2013-06-20 08:05:02' 




38771 

"MetOp-B" 

"2013-06-20 08:05:02" 


Axis index 


LEO-GEO Full Resolution (Tuple) Data for VIIRS vs. GOES13 Event 

Jan. 1, 2013 


a. 


c. 



0.tuple_data.data_0 


Jan. 1, 2013 NPP VIIRS-GOES13 MIIC 1C Event, 

46.4 N, -51.2 S, -114.7 W, -53.2 W, time 2013-01-01T18:49:40Z 
through 2013-01-01T19:17:00Z; a. 1 km GOES13 band 0 data from 
CLASS, b. Matched 2 km VIIRS SDR Radiance_l3_Avg data, c. one 
VIIRS granule (of 7) that make up matched LEO-GEO event; fill values 
at scan angles greater than 31.7° and 56.3° due to "bow-tie" effect. 
Extremely large amount of data for a single event! 


Axis index 

Q.tuple_data.data_0 () 


6025.6 12019.2 18012.8 24006.4 

Data Min - 32.0, Max - 340282346638S2886(MHHH>0Q{H>0{HM>Q{H>0(HM>00.0 


0.tuple_data.Radiance_l3_Avg 


9S9.S 639.S 

Axis index 


Q.tuple_data.RadianceJ3_Avq () 

5.2E+04 


Data Min = 1.4E+02, 


= 6.6E+04 




Axis index 


LEO-GEO Gridded Data for VIIRS vs. GOES13 Event 

Jan. 1, 2013 


histogram_data.data_0_MEAN 



Axis index 


histogram data.data 0 MEAN () 



1.4E+03 5.0E+03 8.7E+03 1.2E+04 1.6E+04 2.0E+04 

Data Min - 1.4E+03, Max - 2.0E+04 



0.0E+00 


histogram_data.RadianceJl_Avg_MEAN () 

9.1E+03 1.8E+04 2.7E+04 3.6E+04 

Data Min = O.OE+OO, Max = 4.5E+04 


GOES13 gridded 


VIIRS gridded 


Gridded data, 0.5 degree, returned from MIIC server-side 2DHistogram function for 
single LEO-GEO matched event; factor of 3500 in data reduction compared to tuple data 
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LEO-LEO Intercomparison for CrIMSS Validation Study (X. Liu) 

Acquire CALIPSO, CrIS, ATMS, VIIRS, and CriMISS Tuple Data - Not Collocated 

IC Event Aug. 12, 2014 



CALIPSO groundtrack for matched event w/ NPP 



Tune MIIC event predictor to use CrIS scan width 


0.tuple_data._AII_Data_CrlS_SDR_AI l_ES_RealLW 



Data Min - 22. S, Max ~ 89.2 


O.tu ple_data._AI l_Data_CrlS_SDR_AI l_ES_Real LW 



Axis index 


Q.tuple_data._AII_Data_CrlS SDR A I I_ES_ RealLW () 

90.0 120.0 

= IS. 9. Max = 12S.S 


Single CrIS SDR LW spectra for FOV1 


Single scan of CrIS LW spectra for FOV1 (30 FORs per scan) 



Server-side Filtering for Surface Site Acquisition 

CERES SSF data, Jan. 2, 2011 
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a. No clear sky filter 




Solar 


b. 5 % clear sky filter 



Deployment Status 


Conducted review with ASDC; presented implementation 
requirements, design, and integration status 

Defined L2 WBS and draft L3 tasks; still need integrated 
schedule with ASDC and NCDC concurrence 

MIIC software is ready for initial deployment (rpms built to 
simplify deployment at NCDC) 

ASDC hardware is in place; need to open up firewall to 
Internet 

NCDC hardware still TBD; planned 30 day dedicated MIIC 
cache + OPeNDAP server 



Issues 


Need integrated schedule 

Need MOU and Interconnection Security Agreement 
with NCDC 

ROSES ACCESS proposal not mechanism for data 
system upgrades (hardware at NCDC TBD) 

EUMETSAT has requested software for evaluation; 
will require NASA Software Release 



